Methods and apparatus for 3d route planning through hollow organs

ABSTRACT

Methods and apparatus assist in planning routes through hollow, branching organs in patients to optimize subsequent endoscopic procedures. Information is provided about the organ and a follow-on endoscopic procedure associated with the organ. The most appropriate navigable route or routes to a target region of interest (ROI) within the organ are then identified given anatomical, endoscopic-device, or procedure-specific constraints derived from the information provided. The method may include the step of modifying the viewing direction at each site along a route to give physically meaningful navigation directions or to reflect the requirements of a follow-on live endoscopic procedure. An existing route may further be extended, if necessary, to an ROI beyond the organ. The information provided may include anatomical constraints that define locations or organs to avoid; anatomical constraints that confine the route within specific geometric locations; or a metric for selecting the most appropriate route. For example, the metric may define the closest route to the ROI such that the route satisfies all applicable anatomical, device, and procedural constraints.

REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/018,953, filed Jan. 24, 2008, which claims priority from U.S.Provisional Patent Application Ser. No. 60/887,472, filed Jan. 31, 2007,the entire content of both of which is incorporated herein by reference.

GOVERNMENT SPONSORSHIP

This invention was made with government support under Grant No.CA074325, awarded by National Institutes of Health. The Government hasrights in the invention.

FIELD OF THE INVENTION

This invention relates generally to medical imaging and, in particular,to a system and method for three-dimensional (3D) route planning andextension to assess hollow, branching organs in patients.

BACKGROUND OF THE INVENTION

Physicians routinely utilize three-dimensional (3D)medical imagingmethods to assess hollow, branching organs in patients. Examples of suchorgans include the liver vasculature, heart vasculature, and the airwaysof the chest.¹⁻³ Examples of 3D medical imaging modalities used toexamine these organs are multidetector computed tomography (MDCT) andmagnetic resonance imaging (MRI).⁴ Often, the physician must find a wayto navigate through the organ to reach a target diagnostic region ofinterest (ROI). The navigation through the organ may be virtual—solelywithin data derived from the 3D image, or it may be by a device, such asan endoscope, moving through the organ in a follow-on liveprocedure.^(1-3,5-7) In each circumstance, an appropriate route throughthe organ to the target location is often difficult to determine.

Lung cancer is the deadliest form of cancer in the United States,accounting for nearly 30% of all cancer deaths and having a five yearsurvival rate of under 15 percents The current state-of-the-art workflowused to assess lung cancer consists of two phases: (1) acquisition andanalysis of 3D volumetric medical images in the form of MDCT scans and(2) bronchoscopy.^(5,9-11)

During phase 1, the physician manually scrolls through a series oftwo-dimensional (2D) axial-plane slices of the 3D MDCT image to identifysuspect nodules and other abnormalities. Upon determining the 3Dlocation and shape of a specific diagnostic region of interest (ROI),the physician determines an appropriate route through the airway tree—acomplex branching structure—to reach the ROI. Defining such a route tosuspect peripheral nodules in the chest can be especially challengingbecause: (1) several airway generations from the trachea need to betraversed, (2) the airway cross-sections are obliquely oriented relativeto the given axial-plane data, and (3) the airway tree continuallybifurcates as a route is traversed.¹²⁻¹³

During bronchoscopy, the physician must translate the mentally definedroute in the static MDCT data to the dynamic video captured by theendoscope and navigate through the airway tree. Even in the bestcircumstances, mentally defining the 3D relationships between theendoscope, airways, and target ROI as depicted in the MDCT data and thevideo feed is difficult. Respiratory motion, coughing, and the inabilityto view ROIs situated beyond the airway walls make the task even morechallenging. Previous research has shown a high degree of variability inbronchoscopy performance between physicians, confirming thesedifficulties.¹⁴

Computer-based image-analysis techniques can help ease theroute-planning task. Image segmentation methods identify image voxelsthat belong to the airway tree.¹⁵⁻¹⁸ Centerline-analysis methodsdetermine the axial structure of the airway tree.¹⁹⁻²⁴ Together, theairway-tree segmentation and centerline-analysis computations provideinputs to 3D visualizations of the airway tree and ROIs. By interactingwith different visualization techniques, a route to an ROI can bemanually defined. However, even with the additional information providedby the centerline-analysis and segmentation computations, manual routedefinition is still a challenging task.

The difficulty of manually selecting an appropriate route is illustratedin FIG. 1, wherein route selection is based on the use of an integrated3D medical image visualization system. The Figure shows two weighted-sumprojections of the chest, with the left projection computed in thecoronal plane and the right projection computed in the sagittal plane.¹⁰The medial axes of the airway tree, derived from an automatedcenterline-analysis method of Kiraly et al.,²⁰ and a peripheral ROI areoverlaid on the projections. The 3D MDCT image size is 512×512×706 withΔx=Δy=0.67 mm, Δz=0.50 mm (case 21405.3a). A route to the ROI ismanually defined by selecting one of the axial paths from the trachea toa peripheral location on the coronal projection. The route is chosenbecause it appears to approach the ROI in this view. However, it is seenin the sagittal view that this route is inappropriate. The airwayterminates in the anterior of the chest, while the ROI is located in theposterior.

The previous example is indicative of the route-planning problem. Evenwith the assistance of 2D, 3D, and quantitative visualization techniquessuch as thin-slab visualizations, 3D airway, surface, and ROIrenderings, and quantitative plots of airway measurements, manual routeplanning is difficult.²⁵⁻²⁸ Centerline-analysis techniques may produceover 200 distinct paths through the major airways. As a result, choosingthe best airway path is a daunting task. Furthermore, airway-treesegmentation methods sometimes miss smaller airways, which often couldbe the ones leading to an ROI. Thus, the extracted paths through theairways can be insufficiently defined to reach a particular ROI.

There has been a great deal of research in extracting centerlines ofbranching anatomical structures from 3D medical images.¹⁹⁻²⁴ Thecenterlines provide the physician with information that would otherwiseneed to be mentally extracted from the volumetric image. They define thetopology of the organ and provide a set of potential routes through it.However, the complex branching organs of interest contain many uniquepaths. Thus, manually searching through a large number of paths to findan acceptable route to an ROI is tedious. Our methods quickly performthis search, determining those paths that provide the most promisingroutes to the ROI. In cases where no appropriate route exists within thepaths as determined by existing centerline-analysis methods, the methodsanalyze the 3D medical image, augmenting the paths with the locations ofpreviously undetected parts of the branching organ.

The most extensive research previously performed in bronchoscopic-devicepath planning is by Kukuk, with the goal of defining a set of parameterssuch as insertion depth, rotation angle, amount of tip deflection, andlength of needle insertion to perform a bronchoscopic procedure.²⁹⁻³¹ Todetermine these parameters, the method precisely models thebronchoscope's configuration within the airway tree as it moves toward apre-determined target position. The method utilizes surfaces derivedfrom an airway-tree segmentation to determine the airway's geometricboundaries. Because the method does not utilize the 3D medical image andinstead relies on the surfaces to represent the airway tree, sections ofthe organ may not be accurately modeled due to imperfections insegmentation techniques. Other inputs include the approximate terminallocation of the endoscope within the airway tree (the route destination)and physical endoscope parameters, such as the endoscope diameter,bending radius, and length of the rigid endoscope tip. Using theseinputs, the method provides the parameters required to navigate to aparticular location. However, the target location is limited to bewithin the original airway-tree segmentation and is required to be knowna priori.

A centerline determination method proposed by Austin included in itscalculations the importance of approaching a target ROI “head-on.”³² Themethod determines paths through sparsely defined centerlines. It is ableto determine the location on a discrete path that is nearest anarbitrary location along the path. The nearest discrete path location ischosen so that the path does not overshoot the target location. Ourproposed methods build on this idea, determining routes to complex,multiple-voxel. ROIs that may extend beyond the original centerlinelocations.

Similar to the method proposed by Austin, Mori et al. describe a routealong an organ's medial axes to a destination that is nearest anarbitrary location in space.³³ In this method, the route is augmentedwith anatomical labels describing the organ segments through which itpasses.

Heng et al. proposed an interactive path-generation method that requiresuser inputs for both the start and end points of a route. By utilizingdynamic programming, this method is able to traverse poor-qualityregions in the image when determining paths. However, in regions of poorimage quality or weakly defined airways, the user may be required toprovide multiple “seed points” to achieve acceptable results. Thismethod, while defining paths robustly, does not fully achieve automatedroute-planning.

Another method to determine bronchoscopic routes has been proposed byGeiger et al.¹³ This method seeks to use the pulmonary blood vessels,which they claim are easier to detect in 3D medical images than theairways themselves, as surrogate pathways to ROI locations. The methodassumes a close correlation between airway locations and vessellocations. However, the existence or accuracy of this correlation is notguaranteed.

Geiger et. al. have also proposed a visualization method that allows aphysician to view the ROI projected onto the airway walls when viewedfrom an endoluminal location.³⁵ This study forgoes automated routeplanning, requiring the physician to know the approximate route(bronchoscope) destination, but aids in needle placement. We have usedsimilar techniques in the past, and visualization tools presentingcomparable information are incorporated into the route validation phaseof our method.¹⁰

Approaches for virtual angiography such as the one proposed by Haigronet al. require little image preprocessing.³⁶ The navigation and routeplanning is controlled by “active vision,” wherein the appearance of thescene at a particular location drives the navigation decisions of themethod. In this approach, segmentation and centerline analysis areessentially done “on the fly,” with the method building up the topologyof the branching structure as it progresses toward a pre-definedterminal point. This method may not be able to determine adequate routesif the organ is difficult to segment, as is often the case in airwaytrees.

Virtual navigation and visualization of the anatomy as depicted involumetric medical images is the topic of much research.³⁷⁻⁴⁰ Often,unlike in the case of bronchoscopy where ROIs must be biopsied todetermine their cellular make-up, virtual endoscopy itself is the endgoal. This is especially the case in virtual colonoscopy, where the goalis to eliminate the follow-on procedure. The work of Kang et al. seeksto define a path (3D locations and viewing directions) through the colonthat is not necessarily an organ centerline. Instead, the path isdirected toward polyp detection.⁴² The method defines paths that areoptimized for the virtual fly-through of the colon. The goal is to showeach section of the colon wall continuously for as long a period a timeas possible so that the physician has an opportunity to detect polyps.In virtual colonoscopy and similar virtual procedures, the methods seekto best define the route to expose ROIs rather than find the best routeto navigate to a specific ROI.

The work of Fujii et al. seeks to find appropriate routes for minimallyinvasive neurosurgery. In this method, voxels within a model brainvolume are assigned local costs. These costs correspond to thedetrimental effects of different surgical events, such as incisions andcontact. The voxel costs are assigned according to the cumulativeknowledge of neurological experts.

In summary, there has been a great deal of research that seeks to define“paths” or “routes” through various organs. These paths/routes may existsolely for virtual interrogation of the image or may be defined in amanner that is conducive for follow-on endoscopic procedures. However,there seems to be no method that finds viable paths to precisely-definedROIs in volumetric images while taking into account the physicalproperties of the endoscopic device, anatomical constraints, andprocedure-specific constraints, and that also allows for the extensionof routes beyond the segmentation, if required.

SUMMARY OF THE INVENTION

This invention resides in methods and apparatus for planning routesthrough hollow, branching organs in patients to optimize subsequentendoscopic procedures. Examples of applicable organs include vasculatureand the airways of the chest, though the invention is not limited inthis regard and may be applied to the digestive tract, the reproductivesystem, ducts, and any other passages.

According to the method, information is provided about the organ and afollow-on endoscopic procedure associated with the organ. The mostappropriate navigable route or routes to a target region of interest(ROI) within the organ are then identified given anatomical,endoscopic-device, or procedure-specific constraints derived from theinformation provided.

The method may include the step of modifying the viewing direction ateach site along a route to give physically meaningful navigationdirections or to reflect the requirements of a follow-on live endoscopicprocedure. An existing route may further be extended, if necessary, toan ROI beyond the organ.

The information provided may include anatomical constraints that definelocations or organs to avoid; anatomical constraints that confine theroute within specific geometric locations; or a metric for selecting themost appropriate route. For example, the metric may devine the closestroute to the ROI such that the route satisfies all applicableanatomical, device, and procedural constraints.

The information may also include the definition of the ROI, or asegmentation of the organ through which navigation will occur in eitherthe 3D image or in the real organ with an endoscopic device. Theinformation may include the central axis or axes of the segmented organas well as a parametric description of the endoscopic device. Theparametric description may, for example, include the diameter,flexibility, or other physical characteristics of the endoscopic device,or descriptions of ancillary devices that may be used in conjunctionwith the primary endoscopic device. The information may be derivedthrough a diagnostic instrument such as a multidetector computedtomographic (MDCT) chest image.

System-level implementations of the invention are also disclosed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows the difficulty of manually selecting an appropriate routein a branching organ;

FIG. 2 is a visual depiction of different structures;

FIG. 3 is a graphical representation of the set of integrated methodsfor automated route planning, including the inputs, algorithms, andoutputs of each method;

FIG. 4 shows some device and procedure-specific constraints;

FIG. 5 shows the value of search distance N in one dimension;

FIG. 6 is a view of the route-planning computer program;

FIG. 7 is a different view of the route-planning computer program;

FIG. 8A shows a view the route in a surface rendered view;

FIG. 8B shows the CT data at this location where the airway diameter is4.1 mm;

FIG. 8C shows an endoluminal view of the airway surface and ROI near theroute destination;

FIG. 8D shows a weighted-sum coronal projection of the CT data with thepaths, route and the ROI;

FIG. 9A shows a 3D representation of the centerlines, extension and ROI;

FIG. 9B is a close-up of the view of FIG. 9A;

FIG. 9C shows a close-up view of the ROI, centerlines, and extension aswell as a slice through which the extension traverses;

FIG. 10 shows a different visual representation of a route with anextended path to reach a peripheral lung nodule ROI; and

FIGS. 11A through 11H show how the view directions may be modified toreach an ROI in the anterior segment of the right upper lobe of thelung.

DETAILED DESCRIPTION OF THE INVENTION

We have developed methods for automated route-planning, drawing upon 3Dimages, that assists follow-on navigation through hollow branchingorgans. In our work, we focus on MDCT images of the chest wherein thegoal is to navigate through the airway tree, often for the treatment andassessment of lung cancer. In this paradigm, we restrict the routes tobe contained within a patient-specific model of a tubular organ such asthe colon or airway tree.

This invention resides in automated techniques for 3D route planningthat: (1) locate the closest paths to a target ROI and (2) extend anincomplete path, if necessary, so that it reaches the ROI. Given a 3Dmedical image, our methods take as input: (1) precisely defined ROIs;(2) a segmentation of the branching organ through with the endoscopicdevice will navigate; (3) centerlines (paths) through the segmentedorgan; and (4) parametric description of the endoscopic device.

We use existing methods for branching-organ segmentation and centerlineextraction. Using these inputs, our methods then (1) identify theclosest routes to the ROI; (2) identify the closest routes that arenavigable given endoscopic-device, procedural, and anatomicalconstraints; (3) extend the existing paths if necessary to complete aroute to the ROI; and (4) modify the viewing orientation at each sitealong a route to reflect a specific visualization or follow-on liveprocedure.

When determining the route to an ROI, we introduce parameters to accountfor possible endoscopic-device, anatomical, and procedure-specificconstraints. To determine the route to an ROI, we find the paths thatsatisfy preset constraints and that are near the ROI. In some cases, thesegmented organ and extracted centerlines do not provide a viable routeto the target. In these circumstances, we extend the existing paths byperforming a directed search from the segmented organ of interest to theROI. The methods have been integrated into a computer-based softwarepackage. Results from human 3D computed tomography chest imagesillustrate the efficacy of the methods.

Methods Overview

In this section, we describe our approach to the route-planning problem,with a focus on finding routes through chest airways. We begin with adiscussion of the representation of the branching organ and its medialaxes, the inputs to the methods, and the outputs. We then provide anoverview of our set of integrated methods.

Branching Organ and Route Representation

The determination of a route through an organ to an ROI requires a 3Dgray-scale medical image I of the organ and ROI. From I, animage-segmentation routine (e.g., that of Kiraly et al.¹⁶) generates abinary 3D image Is that defines the organ of interest. We use theconventions of Kiraly et al. to describe the medial axes of Is²⁰

Collectively, the set of all medial axes comprise a tree, T=(V,B,P),where V={v1, . . . , vL}, is the set of view sites, B={b1, . . . bM} isthe set of branches, P={p1, . . . , pN} the set of paths, and L, M, andN are integers ≧1. A view site v=(s,d) is the data structure thatdiscretely represents the medial axes, with s corresponding to the viewsite's 3D location (x, y, z), and d its orientation. The orientationcontains two direction vectors: d=({right arrow over (T)}, {right arrowover (U)}). {right arrow over (T)} is the tangential view direction and{right arrow over (U)} is the up direction. A branch, b={va, . . . ,vl}, va, . . . , vl∈V, combines connected view sites between the organ'stopologically significant points. Topologically significant pointsinclude the origin (root) of the organ, points where the organbifurcates, and the terminating points. A branch must contain two andonly two topologically significant points that define the beginning andend of the branch, respectively. A path, p={ba, . . . , bm}, ba, . . . ,bm∈B, contains a set of connected branches. Paths must begin at a rootbranch b1 and terminate at the ends of Is.

We augment these data structures with the route data structure r={vA, .. . , vD}, with some v∈V and others new, which consists of a collectionof connected view sites. The final view site along the route, vD, is theroute destination, which is located near the ROI. If an adequate routeis not found within the original tree, our set of integrated methodsextends the definition of the organ beyond Is by appending additionalview sites to the original tree. The appended view sites, together withview sites within the original tree, define a new path that terminatesnear the ROI. We refer to the newly-defined sites along the path as thepath extension.

FIG. 2 gives a visual depiction of these different structures. Theoriginal organ segmentation is contained within the smooth boundary. Themedial-axes tree is represented by the circular view sites within thesegmentation. ROIs are shown as green hexagons. The route to the ROI atthe top of the figure is entirely contained within the original tree. Atthe bottom of the figure, the route to the ROI requires path extension.

Inputs

The inputs required by the set of integrated methods are given below:

-   -   1. 3D medical image data of the anatomy of interest. For        example, a 3D MDCT image.    -   2. Predefined diagnostic region of interest (ROI) in the 3D        medical image data. The ROI may be defined manually,        semi-automatically, or automatically. Examples of ROIs could be        suspect peripheral cancer nodules, central chest lymph nodes,        infiltrates, or other suspicious chest masses.    -   3. Pre-segmented region of the anatomy of interest. For example,        the segmented airway tree.    -   4. Automatically precomputed tree. As an example, the tree could        be the central axes of the lung airways. Each view site in the        tree can optionally be augmented with anatomical labels that        describe its location within the anatomy. For the lung, these        labels could give lobe and segment assignments.    -   5. Requirements on routes that are procedure-specific. As an        example, a requirement may be that the endoscope's orientation        at the route destination is such that instruments can extend        from the endoscope's working channel toward the ROI.    -   6. Dimensions and mechanical bending properties of an endoscope.        The route computation may use endoscope constraints for        computing feasible trajectories of a real endoscope for later        interventional endoscopy.    -   7. Anatomical requirements. An example of an anatomical        requirement may be that candidate routes do not terminate at a        location and orientation where the interventional endoscopic        procedure may puncture major vessels or other sensitive        anatomical regions.

Outputs

The set of integrated methods outputs the N closest routes to the ROI.Each route consists of a sequence of view sites within the 3D medicaldata. A route begins at the base of the anatomy of interest, such as thetrachea for the airway tree, and concludes as close to the ROI surfaceas possible. The viewing directions along the route can optionally bemodified so that each view site is oriented as it would be in a livefollow-on procedure. A graphical representation of the set of integratedmethods for automated route planning, including the inputs, algorithms,and outputs of each method is given in FIG. 3. The inputs to thealgorithms of each method are given in parallelograms. The algorithms ofeach method are contained in rectangles. The enumeration of thealgorithms is consistent with their presentation later in this section.The rounded shape at the right of each method gives the method's finaloutput.

The Integrated Methods

This section provides an overview of the set of integrated methods.Specific algorithmic and mathematical details for each method arepresented below. The objective of each component is summarized asfollows:

-   Method 1: Determine the routes that are closest to the ROI. Find the    routes contained within the input tree, T, with destinations nearest    the ROI.-   Method 2: Find the route that is closest to the ROI and also    satisfies device, procedural, and anatomical constraints. Find the    routes contained within T that are navigable for a particular    device, terminate at acceptable locations and have destinations    nearest the ROI.-   Method 3: Find undetected paths. If the closest routes contained    within T do not adequately reach the ROI, perform a directed search    for undetected segments of the organ of interest within I.-   Method 4: Find undetected paths near constraint-satisfying    locations. Perform a directed search for undetected segments of the    organ of interest from locations that satisfy device, procedural,    and anatomical constraints.-   Method 5: Determine procedure-appropriate viewing directions.    Compute appropriate navigation directions at each view site along    the determined route to reflect the procedure to be performed.

The remainder of the section provides more detail for each of themethods.

Method 1: Stay within the Original Tree

This method finds the routes that are closest to the ROI, with the routesearch restricted to the set of view sites V in the input organ tree T.The closest path to the ROI pc is found first. The route destination vDis the view site on pc that is nearest the ROI. In many cases, thissimple method yields acceptable routes with minimal computationalcomplexity. To better the chances of finding an acceptable route,multiple routes are found. The method outputs the N closest routes, eachwith destinations on unique tree components: view sites, branches, orpaths.

To find the closest routes, the distance from each 3D view site to theROI is determined. The ROI, like the paths, is represented by acollection of discrete 3D points (voxels) in space. The method finds theROI voxel that has the smallest Euclidean distance to each discreteviewing site. The distance calculation accounts for the anisotropicnature of many 3D volumetric medical images.

Each view site, containing information about the branch and path towhich it belongs and the minimal distance to the ROI, is pushed onto apriority queue. The N closest sites are found by popping values off thepriority queue in ascending distance from the ROI until N sites arefound on unique segments. For performance reasons, distances arecalculated only for those voxels on the ROI surface. The procedure fordetermining routes in this manner is outlined in Algorithm 1.

Algorithm 1: Closest Routes. Procedure for finding the N closest routes,each with destinations in unique tree components. Data: Organ tree T,list of ROI voxels; Result: Ordered list of closest routes; forall ROIVoxels do  | Remove from consideration all ROI voxels not neighbored onall sides by ROI voxels; end forall View sites in the organ tree T:v_(i), (i = 1, . . . ,L) do  | forall Considered ROI voxels do |  | Compute distance to view site v_(i);  |  | if Current distance isless than minimum distance then  |  |  | Set minimum distance to currentdistance;  |  | end  | end  | Push minimum distance and path informationonto priority queue; end while N view sites an different tree componentsnot found do  | pop closest item from priority queue;  | if Popped viewsite on previously unseen tree component then  |  | Add view sites fromroot to the popped view site to the ordered    output list;  | end end

Method 2: Satisfy Additional Physical Constraints

The absolute closest routes found by Method 1 may yield results that aresuboptimal for either virtual inspection of the 3D data or possiblefollow-on live procedures. The closest route destinations are oftenlocated such that the ROI is nearly at a right angle to the tangentialview direction at the route destination. As such, the ROI may not bevisible in virtual techniques or reachable by tools that extend straightout of the tip of the endoscopic device. In other cases, the routedestination may be unreachable by endoscopic devices if it is locatedbeyond organ constrictions or if the route has too great a curvature. Toalleviate these issues, this method builds on Method 1 by requiring thatcandidate routes satisfy constraints related to the physicalcharacteristics of the endoscopic device used, the procedure to beperformed, and the anatomy. These restrictions include:

1. Endoscope Diameter

-   -   All view sites along an acceptable route must be above a minimum        diameter threshold. If the procedure requires an endoscopic        device of a certain size be inserted into the organ, this        constraint ensures the device can fit within the organ along the        entire route.

2. Branching Angle

-   -   Every view site along an acceptable route must have a branching        angle less than or equal to the branching angle threshold. If        the procedure requires an endoscopic device of a certain        flexibility to be inserted in the organ, this constraint ensures        that the device can bend through the route.

3. ROI/View Site Location Geometry

-   -   A viable route destination is one in which a sufficient fraction        and/or minimal volume of ROI voxels lie within the “diagnostic        field of view.” If an endoscopic device needs to interact with        the ROI (such as a needle biopsy), this constraint ensures that        the device approaches the ROI in such a manner that the        interaction is possible.

4. Anatomical Restrictions

-   -   The route destination should be at an appropriate location        within the organ. For instance, a constraint on bronchoscopic        procedures may be that the destination be at a location and have        an orientation that precludes puncturing major blood vessels.        Another constraint may be that the destination is located in the        same lobe of the lung as the target ROI. If the procedure        requires a needle or other type of tool to interact with the        ROI, the latter constraint prevents puncturing a lobe of the        lung when extending the tool from the route destination to the        ROI.

FIG. 4 illustrates some device and procedure-specific constraints.Section (A) shows a simple model of the endoscope tip. The three mostimportant characteristics of the device that determine its ability toreach a specific route destination are its diameter (it may not be ableto fit past constrictions or in small peripheral locations of theorgan), its bending angle (it may not be able to negotiate through sharpturns) and the length of the tool that extends out of the device'sworking channel. The length of the tool determines how close routes mustbe to the ROI. Section (B) illustrates the need to approach the ROI“head on,” which is quantified by a diagnostic field of view, located atthe endoscope tip. Section (C) shows a cross section of the branchingorgan. The organ is not necessarily cylindrical, especially atbifurcation regions. As such, the “minimum diameter” gives a measure ofwhether the endoscope can fit at a particular location. Finally, section(D) shows the endoscope (blue) at a location between branches with alarge branching angle. If the endoscope is not flexible enough, it maynot be able to move through these high-curvature regions.

Algorithm 2 outlines the procedure to determine candidateconstraint-satisfying route destinations. The output of this algorithmbecomes the input to Algorithm 1 which then finds the N closest routes,each with destinations on different tree components.

Algorithm 2: Sites Meeting Constraints. Procedure for eliminating viewsites which are inaccessible given device and anatomical constraints, orhave orientations relative to the ROI that are unacceptable forperforming a specific procedure. The view sites that remain after thesecalculations are candidate route destinations. Data: Set of view sites Vin organ tree T, list of ROI voxels: Result: Set of candidate routedestinations C, with C ⊂ V, that meet constraint restrictions; Set C toV; forall Unique input paths: p_(i), (i = 1, . . . , N) do  | forallView sites: v_(j), (j = 1, . . . , M) on p_(i) do  |  | if View sitev_(j) does not meet anatomical restrictions then  |  | | Remove Viewsite v_(j) from output set C  |  | else  |  |  | if diameter of v_(j)less than minimum allowed OR branching  |  |  | angle greater thanmaximum allowed then  |  |  |  | Remove view site v_(j) and all viewsites below it  |  |  |  | (all v_(x) on p_(i) such that j ≦ x ≦ M) fromoutput set C;  |  |  | else  |  |  |  | Set acceptable ROI count to zeroforall ROI Voxels do  |  |  |  |  | if Angle between ROI voxel and v_(j)less than minimum        angle then  |  |  |  |  |  | Increment ROIcount;  |  |  |  |  | end  |  |  |  | end  |  |  |  | if ROI count notabove absolute/fractional threshold then  |  |  |  |  | Remove v_(j)from output set C;  |  |  |  | end  |  |  | end  |  | end  | end end

Method 3: Extend Existing Paths to ROI Neighborhood

In some circumstances, the set of paths in the input tree may not yieldroutes that adequately reach the ROI. Due to imperfections in the 3Dimages and variations in patients' conditions, segmentation proceduresmay not be able to completely define the organ of interest. Because thecenterline-analysis methods extract the organ's tree from the organsegmentation, imperfections in the segmentation result in an incompletetree. A more detailed explanation of the necessity of expanding the treebeyond what is captured by centerline-analysis methods to findappropriate routes to ROIs and the mathematical details of this methodare presented elsewhere herein.

In cases where the existing paths are inadequate, this method extendsthe paths toward the ROI surface. A view site in the extended path maybe a member of the original tree (v∈T) or the extension (v∉T).Extensions are computed by comparing an individual voxel's localappearance to that of the organ of interest. Each voxel is assigned acost, with low costs assigned to those voxels whose appearance issimilar to the organ and high costs assigned to voxels who do not looklike the organ. A path extension is found by searching for alow-cumulative-cost set of voxels that lead from the ROI to the existingsegmentation. The local costs are determined using the followingcriteria:

1. Gray-Scale (HU) Value

Voxels within an airway should have low HU values corresponding to air.

2. Local Valley Behavior

The voxels within an airway should be within a local maxima, due to thelarger HU values of surrounding airway walls.

Each voxel in the volumetric image is represented as a vertex in adirected graph. Directed graph edges connect each voxel to its 26neighbors, with edge weights E(M, N) from voxel M to its neighbor Ndetermined by the local vertex cost scaled by the Euclidean distancebetween voxels (vertices).

The first step of the method is to dilate the ROI. The amount ofdilation corresponds to the distance an extended path may terminate fromthe ROI. Because ROIs often do not lie within the airways, ROI expansionprovides an opportunity for the ROI to intersect previously undetectedairways.

In a manner similar to others previously proposed, Dijkstra's algorithmdetermines the minimal cumulative-cost set of 26-connected voxels fromthe dilated ROI to the existing airway tree.^(34,43-45) The algorithmstarts at a location on the dilated ROI, whose voxels have zero localcost. It then iteratively expands out from the vertex with lowestcumulative cost, summing the edge weight between the lowestcumulative-cost vertex and each connected vertex. The connected vertex'scumulative cost is set as this value unless the connected vertex has alesser previously-determined cumulative cost. The algorithm keeps trackof the edges and vertices traversed to get to each expanded location.These edges and vertices are the lowest-cost connected set of voxelsfrom the ROI to a given location. Expansion halts when the vertex to beexpanded is on one of the existing 3D paths. See Algorithm 3.

Algorithm 3: Extend Paths to ROI. Procedure for extending a path to thevicinity of an ROI. The algorithm uses Dijkstra's algorithm to traversea weighted, directed graph derived from the 3D medical image. Thisalgorithm returns a set of connected voxels (3D locations) that extendfrom the existing tree to a terminal location near the ROI. Data: 3Dmedical image I, set of ROI voxels, organ tree T; Result: Lowest-costconnected set of 3D voxel locations connecting ROI to a path in T;forall Voxels in ROI that are not completely surrounded by other ROTvoxels do  | Add to ROI definition those voxels not already contained inROI and within neighborhood distance; end Choose a location L on ROI tobegin, set cumulative cost at L to zero, back pointer to nothing; Push Lonto priority queue; while Location of minimum cumulative cost voxel Min priority queue not on a path do  | Remove M from priority queue; | forall 26 Neighbors of M not previously placed in queue do  |  | ifCumulative cost of current neighbor not previously set then  |  |  | Setcurrent neighbor N's cumulative cost to M's cumulative cost + E(M,N) andback pointer  |  |  | of N to M;  |  | else  |  |  | if Cumulative costto M + E(M,N) is less than current cumulative cost to N then |  |  |  | Update cumulative cost of N to (M cumulative cost + E(M,N))and back pointer of N to  |  |  |  | M;  |  |  | end  |  | end  | endend

The path extension of Algorithm 3 is often jagged, which is usuallyinconsistent with the smooth underlying anatomical structures.Furthermore, the connected set of 3D locations output by Algorithm 3lack viewing directions. These concerns are addressed by fitting a setof 3D lines to the points. The 3D lines well-approximate short,low-curvature peripheral branches. Because the path extension maytraverse multiple underlying anatomical branches, multiple 3D linesegments are fit to the voxels.

Combinations of voxels in the connected set define end points ofpotential line segments. The set of line segments with the bestcumulative “line-fit score” is accepted as the solution. The connectedset of voxels is projected onto these line segments, with the tangentialviewing directions pointing away from the existing path locations andtoward the ROI. The up viewing direction is projected from the voxellocated on the original centerline to each successive voxel toward theROI. These steps are detailed in Algorithm 4.

Algorithm 4: Smooth Path Extension, Determine Viewing Directions.Procedure for smoothing the 3D path extension generated by Algorithm 3.In this algorithm, the 3D locations of view sites along the extendedpath are smoothed and viewing directions are assigned. Data: Connectedset of 3D locations, connecting view site on existing path; Result:Smoothed Set of view sites with associated viewing directions; Set bestline fit score to ∞; forall Combinations of K line segments. whoseendpoints are voxels within the connected set do  | if Currentcombination's line fit score less than best line fit score then |  | Set line fit score to current score;  |  | Retain currentendpoints;  | end  | forall Voxels in connected set, in order, startingalth voxel added last in Algorithm 3 do  |  | Add to output list 3Dlocation projection;  |  | of voxel onto retained line segments; |  | if First voxel on connected set then  |  |  | Project up viewingdirection {right arrow over (U)} from connecting view site on existingpath;  |  | else  |  |  | Project {right arrow over (U)} from previouslocation;  |  | end  |  | Set tangential viewing direction vector {rightarrow over (T)} along retained line segment;  | end end

Method 4: Satisfy Physical Constraints and Incorporate Path Extension

This method is a hybrid of Methods 2 and 3 wherein the path extensionprocedure of Method 3 is modified so paths are extended from thoselocations that satisfy the device, procedural, and anatomicalconstraints. As such, the endoscopic device can reach at least thebeginning point of the extension.

To determine this path extension, the inputs and outputs of Method 2 areused as inputs to Method 4. Voxels in the segmentation Is are eliminatedfrom consideration if they are closer to inaccessible view sites thanaccessible view sites. The distances required to make this determinationare efficiently computed using the distance transform with a Euclideandistance measure.⁴⁶ Starting with those view sites that remain afterMethod 2, the method incrementally finds the distance from each viewsite to each voxel in Is. This process stops once the only voxels whosedistances have not been calculated are at a distance that is at least asgreat as the diameter of the largest airway in Is.

The process is repeated for the inaccessible view sites, keeping trackof all voxels added who have not had distances set or have a distancefrom an eliminated view site that is less than the distance set by theprevious pass. These voxels must not be contained within pathextensions. Using the output of this method, the local vertex costs ofMethod 3 are set to values that are effectively infinite for theblack-listed voxels so that the path extensions will not pass throughthem. With this modification, path extensions are found using theprocedure described in Method 3. The procedure to determine theblack-listed voxels is outlined in Algorithm 5.

Algorithm 5: Extensions From Acceptable Locations. Procedure fordetermining the set of black-listed voxels so that a path extension doesnot connect to inaccessible regions of the organ tree. Data: Originaltree T, pruned tree that satisfies constraints T_(P), organ segmentationI_(S); Result: Set of black-listed (eliminated) voxels E through whichthe path-extension cannot pass; forall View sites meeting constraints(v_(i) ∈ T_(P)) do  | Push into priority queue with distance of 0; | Set view site location as visited with nearest centerline locationpointer as itself; end while Distances popped from top of queue are lessthan maximum centerline diameter do  | Determine distance from originalview site location to each of 26 neighbors in I_(S);  | if Neighbordistance has not been computed or distance from original view site topopped voxel is less  | than popped voxel's current distance then |  | Set neighbor's pointer to popped view site's original location; |  | Set distance of neighbor from pointed to location;  |  | Pushneighbor on priority queue;  | end end Flush the queue; forall Viewsites not meeting constraints (v_(i) ∈ (T \ T_(P))) do  | Push intopriority queue with distance of 0;  | Set view site location as visitedwith nearest view site location pointer as itself; end while Distancespopped from top of queue are less than maximum view site diameter do | Determine distance from original view site location to each of 26neighbors in I_(S);  | if Neighbor distance has not been computed ordistance from original view site to popped voxel is less  | than poppedvoxel's current distance then  |  | Set neighbor's pointer to poppedview site's original location;  |  | Set distance of neighbor frompointed to location;  |  | Push neighbor on priority queue;  | end endforall Voxels in segmentation I_(S) do  | if Voxel points to view sitenot meeting constraints then  |  | Add voxel to E  | end end

Method 5: Incorporate Procedure-Specific Navigation Directions

The methods presented thus far, as well as previously proposedcenterline-analysis methods, do not necessarily give physicallymeaningful navigation directions. As an example, it may be desirable forthe navigation directions to reflect accepted bronchoscope orientationsat locations along a route. Typically, the navigation directions arechosen so that the tangential viewing direction (the direction in whichthe view site is looking) faces toward the end of a branch. Because mostendoscopic devices are tubular objects within larger tubular organs,they are generally oriented in the direction of the organ. However,since the endoscopic devices are free to rotate, the viewing directionsmust also define the relative “up” direction at each view site.Generally the up vector, which quantifies this up direction, isarbitrarily chosen. However, in endoscopic procedures, such asbronchoscopy, there is often an accepted practice for the rotation ofthe endoscopic device so that the physician is able to maintain a senseof 3D orientation throughout the procedure and smoothly navigate along aspecific route. This method adds a refinement to the routes derived byMethods 1-4 that assigns practical, useful directions to the route viewsites.

Viewing directions are assigned to reflect live bronchoscopy standardpractices. An example of rules used to orient the bronchoscope is givenbelow. In this procedure, directions are determined in three differentways:

1. Anatomically Accepted Standard Positions. During the first few airwaygenerations, the bronchoscope is oriented in standard positionsdepending on its location within the anatomy.

2. Orient the Route Up in Periphery. As the bronchoscope moves towardthe lung periphery, there are no longer standard orientations linked tospecific locations in the anatomy. Instead, if a “sharp” turn isrequired at a bifurcation the endoscope is rotated so that the nextbranch along the route is located at the top of the scene.

3. Orient Toward ROI at Route Destination. The bronchoscope is orientedso that the endoscopic tool can extend from it to interact with the ROIat the route destination. Because the tool extends from one side of thebronchoscope (not the center), the orientation is such that the ROIcenter of mass and the endoscopic tool are located in the same portionof the view.

Viewing directions that reflect these rules are determined withknowledge of the geometry and anatomical make-up of the route. If aviewing site is near the end of a branch or the route destination, itsorientation is modified to reflect the desired orientation going intothe next branch or the destination. The modifications happen graduallyover several view sites so the transitions are smooth and reflective ofhow the device moves. Algorithm 6 describes this process.

Algorithm 6: Route-Specific Viewing Directions. Procedure fordetermining viewing directions along a route for a specific endoscopicprocedure. Data: Organ tree T, route r, target ROI; Result: Appropriateviewing directions along route r; forall View sites along r: v_(i),(i =1, . . ,D) do  | Determine tangential direction {right arrow over (T)}; | if First view site an route (v₁) then  |  | Set initial up direction{right arrow over (U)};  | else  |  | Project {right arrow over (U)}form previous view site;  | end  | if View site within N view sites ofend of current branch then  |  | if Anatomical rules exist for enteringnext branch then  |  |  | Determine proper up viewing direction enteringnext (child)      branch {right arrow over (U)}_(C);  |  | else |  |  | Determine {right arrow over (U)}_(C) using general rules; |  | end  |  | Blend current site's {right arrow over (U)} with {rightarrow over (U)}_(C);  | end  | else if View site is within M view sitesof route destination v_(D) then  |  | Determine appropriate orientationd_(D) at v_(D);  |  | Blend current orientation, d_(C), with d_(D); | end end

Method Detail and Implementation

Method 1: Stay within the Original Tree

This method gives a route that stays completely inside a known,previously segmented airway. This implies that an endoscope, if smallenough and flexible enough, can follow the route. This route does notnecessarily completely reach the ROI, but a needle could perforate theairway wall to biopsy the ROI.

Rather than finding a single route that is the absolute closest to theROI, the method provides the user with the set of N closest routes.However, the N absolute closest destinations will often be neighbors ofone another. If the closest route destination is located on a path thatyields an unacceptable route, its neighbors on the same path will likelydetermine unacceptable routes as well. Therefore, the method returns theN closest routes, each with destinations on unique tree segments: viewsites, branches, or paths.

Method 2: Satisfy Additional Physical Constraints

This method gives a physically traversable route, via the endoscope, forreaching the ROI during a live procedure.

Details about the parameters and restrictions used are provided below:

1. Endoscope Diameter

Devices of a given diameter may not reach all parts of the airway treewithout causing harm to the patient. On a valid route, all of the viewsite locations between the route origin and destination are at least aswide as the diameter of the endoscopic device. Airways, while beingtubular structures, in general do not have circular cross sections. Thisis especially true in the case of stenosed airways and at bifurcations.Whether an endoscope can fit at a given view site is thereforedetermined by the minimal “diameter” of the cross-section of the viewsite.

2. Branching Angle

Endoscopic devices have a limited amount of flexibility. Therefore, thedevice may not be able to navigate through sharp turns. As the sitesalong any one particular branch tend to have little curvature, the mostproblematic regions are those in the transition from one branch toanother. The branching angle formed between consecutive branchesaccounts for the flexibility constraints. By definition, locations onthe same branch have a branching angle of zero.

3. ROI/View Site Location Geometry

The view sites nearest the ROI, even those on separate branches orpaths, may be impractical for performing specific procedures. Manyprocedures require the route to approach an ROI “head-on.”Quantitatively, the angle formed between the view site and voxels in theROI and the tangential viewing direction should be small. By doublingthe maximum angle allowed between the tangential viewing direction{right arrow over (T)} and the direction formed by the vector connectingthe ROI location to the view site location, a component of the“diagnostic field of view” is defined. This “diagnostic field of view”is a cone whose tip is located at the view site, is oriented along{right arrow over (T)}, and the aforementioned angle defines the anglebetween the sides of the cone. A valid route destination is thereforeone in which an acceptable amount (either fractional or absolute, asspecified by the user) of ROI volume lies within the cone. FractionalROI volume is determined by the ratio of the number of ROI voxels thatlie within the cone to the total number of ROI voxels. Absolute ROIvolume is determined by summing the number of ROI voxels that fallwithin the cone and scaling the result by the anisotropic voxeldimensions.

4. Anatomical Restrictions

Often, anatomical restrictions dictate where a route should terminate.In many circumstances it is required that route destinations not belocated near major blood vessels or other sensitive regions to minimizethe risk of harm to the patient. This type of anatomical restrictionindicates regions where the route should not terminate. Conversely,restrictions may dictate where routes should terminate. As an example,the ROI and route destinations may be required to be within the samelobe of the lung. The motivation for this requirement is to prevent lungperforation or other damage during a procedure (such as a biopsy). Theanatomical information for the ROI, such as the lobe in which it islocated, is assigned when it is segmented. Identification of the majorairways and other airway-tree anatomical labeling tasks can bedetermined by automated techniques.^(33,47,48)

Method 3: Extend Existing Paths to ROI Neighborhood

The routes determined by Method 1 may fail to adequately approach theROI because Method 1 utilizes paths derived from a global segmentationof the branching organ. Airways appear in MDCT images as dark voxelswith nominal Hounsfield Unit (HU) values of −1000 surrounded by thebrighter airway wall with correspondingly larger HU values. Segmentationmethods try to find all of the voxels contained within valid airways.However, efficient segmentation of the entire branching structure is adifficult task due to image noise, partial-volume effects, and otherimage artifacts.

Our method utilizes input segmentation methods that are designed tocapture as much of the airway tree as possible without “leaking” outsideof airways into the rest of the lung. For a route-planning method,leakage is more troublesome than not detecting some airways as routesmay be proposed that navigate through unreachable locations. Aconsequence of conservatively defining the airway tree is that althoughinput segmentations have very low false-positive rates (classifyingnon-airway voxels as airway voxels), they may exhibit relatively highfalse-negative rates (not correctly identifying airway voxels). Therates of false-negatives are especially high in small-diameterperipheral branches where the airway wall is thin and the few voxelslocated within the airway are easily corrupted by partial-volumeeffects—wherein a voxel's HU value is averaged when it straddles unliketypes of matter such as the airway wall and air—and noise. Due to thishigh rate of false-negatives, path extension is often required.

Dilation of the ROI prior to the extension gives the ROI the opportunityto intersect the organ of interest. Without this dilation, the pathextension would have to jump out of the previously undetected segmentsof the branching organ and eventually intersect the ROI. The voxelsencountered as the extension bridges the gap between the true underlyingorgan and the ROI can have a significant impact on the final lowest-costpath. In dilating the ROI by an appropriate amount, as determined by thesize of the ROI, the procedure performed, and the medical instruments tobe used during the procedure, these problems are mitigated.

Voxel Cost Determination

The local cost determination is application-specific. For instance, thelocal appearance of pulmonary vessels is quite different than pulmonaryairways. Blood vessels are more homogeneous than airways and are thuseasier to detect. Because we are interested in finding airways, thefollowing cost measures are tailored accordingly. However, anylocally-determinable quantification of the appearance of the organ ofinterest could be used instead.

Local airway costs are assigned such that voxels whose local appearanceis indicative of belonging to an airway are assigned low costs.Likewise, those voxels who do not have the local appearance of an airwayare given high costs. Voxels that belong to the dilated ROI are assignedzero cost. Using the qualitative properties of airways previouslydescribed, the quantitative cost of a voxel V at 3D locationX=[x,y,z]^(T) is denoted as C(X). This cost is determined as:

$\begin{matrix}{{C(X)} = \{ {\begin{matrix}{0,} & {{{if}\mspace{14mu} X} \in {ROI}_{dilated}} \\{{L(X)},} & {else}\end{matrix}.} } & (1)\end{matrix}$

The local cost L(X) of a non-ROI voxel is given by:

$\begin{matrix}{{L(X)} = \{ {\begin{matrix}{1,} & {{if}\mspace{14mu} {f_{G}(X)}1} \\{1,} & {{{if}\mspace{14mu} {f_{V}(X)}} = 1} \\{{{{wf}_{G}(X)} + {( {1 - w} ){f_{V}(X)}}},} & {else}\end{matrix}.} } & (2)\end{matrix}$

The function ƒG(X) determines the component of the cost based solely onthe voxel's HU value, ƒV(X) is a “valley” cost enforcing the requirementthat an airway voxel is should be less than the surrounding airway wallsand determines the relative weighting given to each.

The gray-scale HU cost ƒG(X) enforces the fact that airway voxels musthave small HU values,

$\begin{matrix}{{f_{G}(X)} = \{ {\begin{matrix}{0,} & {if} & {{I(X)} \leq {- 1000}} \\{( \frac{{I(X)} + 1000}{T + 1000} )^{2},} & {if} & {{{- 1000} < {I(X)} \leq T},} \\{1,} & {if} & {T < {I(X)}}\end{matrix}.} } & (3)\end{matrix}$

I(X) is the gray-scale value of the image volume at a location X and Tis the maximum gray-scale threshold for voxels to be considered air andis nominally set to a value of −750HU.

The function ƒV is the median “valley” cost which enforces that thevalue of an airway voxel must be smaller than those in its neighborhood.As an intermediate step, the value

$\begin{matrix}{{f_{MED}(X)} = {{{MED}\begin{pmatrix}{{{MIN}( {{MAX}( {{I( {X \in N_{x}^{-}} )},{I( {X \in N_{x}^{+}} )}} )} )},} \\{{{MIN}( {{MAX}( {{I( {X \in N_{y}^{-}} )},{I( {X \in N_{y}^{+}} )}} )} )},} \\{{MIN}( {{MAX}( {{I( {X \in N_{z}^{-}} )},{I( {X \in N_{z}^{+}} )}} )} )}\end{pmatrix}} - {I(X)}}} & (4)\end{matrix}$

is computed. Equation 4 determines if a voxel at X is smaller than itsneighbors, where N is the search distance along a direction x,y,z. Inone dimension, this value is illustrated in FIG. 5. MAX I(X∈N⁺), as theminimum of the maxima on either side of the voxel at location X, is thevalley cost for this dimension.

The median value is then linearly scaled by the function

$\begin{matrix}{{f_{V}(X)} = \{ {\begin{matrix}{1,} & {if} & {{f_{MED}(X)} < T_{L}} \\{{\frac{1}{T_{L} - T_{H}}( {{f_{MED}(X)} - T_{H}} )},} & {if} & {T_{L} \leq {f_{MED}(X)} \leq T_{H}} \\{0,} & {if} & {T_{H} < {f_{MED}(X)}}\end{matrix}.} } & (5)\end{matrix}$

The values TL and TH are the upper and lower thresholds that determinethe range of acceptable valley-cost values.

Connected Set Determination

Path-extensions are determined by constructing a graph where each voxelin the medical image is a vertex with edges connecting each voxel to its26-connected neighbors. A directed edge weight E(U, V) from a voxel atlocation U to one at location V is determined by

E=(U,V)=|U−V|C(V),  (6)

the distance between the voxel locations scaled by the voxel cost forthe voxel at location V.

Smooth 3D Locations and Determine Viewing Directions

The output of Algorithm 3 is typically jagged. Smoothing these locationsyields path extensions that are more visually appealing. The smoothingis accomplished by fitting sets of line segments, the endpoints of whichare defined by combinations of voxels, each at a location (UJ), withinthe jagged connected set (J). The best set of end points (S) is thatwhich minimizes the total error (E) given by:

$\begin{matrix}{{E = {\sum\limits_{U_{J} \in J}( {{\alpha {{U_{J} - {{Proj}( {U_{J},S} )}}}^{2}} + {\beta \; {L( {{Proj}( {U_{J},S} )} )}}} )}},} & (7)\end{matrix}$

which is the sum of the squared distance between the voxels in theconnected set and their projection onto the line segments defined by Sscaled by a weighting factor and the gray-scale cost of the projectionof the point scaled by another weighting factor. By defining the errorin this manner, the projections of the voxels in the connected set mustbe geometrically near the original voxels as well as in locations thathave the local appearance of airway voxels.

Method 5: Incorporate Procedure-Specific Navigation Directions

This method modifies the orientation of each view site to reflect adesired procedure or visualization. Two orthogonal sets of directionsare calculated at each view site: the tangential viewing direction andthe up direction. The tangential viewing directions are computed usingKiraly's method so that each view site faces the end of its branch.²⁰This method linearly blends the first N tangential viewing directions ona branch with the tangential viewing direction at the last view site onthe previous (parent) branch to avoid abrupt tangential viewingdirection changes at bifurcations. To determine the up direction at eachsite, Kiraly's method selects an up vector at the root which isprojected from site to site along each path. Each projection seeks tominimize the change in the up direction between contiguous sites whilesatisfying the constraint that the up vectors must be orthogonal to thetangential vectors.

Our procedure-specific navigation directions modify thearbitrarily-defined up vectors to correspond with the orientation of theendoscopic device during a procedure. Often, the up direction isdependent upon the relative orientation of the next branch (the child,bC) to be encountered along a route as compared to the current branch(the parent, bP). Each branch has a direction vector, {right arrow over(C)} and {right arrow over (P)}, respectively. {right arrow over (C)}and {right arrow over (P)} are oriented at the first view site on theirbranches and point toward the last view site on their branch. Byconvention, {right arrow over (C)} and {right arrow over (P)} are unitvectors. We use these directions to determine the up vector, {rightarrow over (U)} and the tangential vector, {right arrow over (T)}, alsounit vectors, with {right arrow over (U)} orthogonal to {right arrowover (T)} at each view site. The orthogonality constraint requires theup vector to lie in the plane whose normal is given by {right arrow over(T)}. To determine procedure-specific viewing directions, the projectionis denoted by {right arrow over (R)}. Because the orientation of thedevice is often dependent upon the orientation of the next branch, bycalculating {right arrow over (R)}, the viewing directions can beconstructed so that up vector at the final site is oriented in anydesired position relative to {right arrow over (R)}. {right arrow over(R)} is determined by:

{right arrow over (R)}=−(({right arrow over (P)}×{right arrow over(C)})×{right arrow over (P)}).  (8)

The view site is then determined by setting {right arrow over (U)} and{right arrow over (T)} such that {right arrow over (U)}'s relativeorientation compared to {right arrow over (R)} abides by the rules forthe route. In the example that requires the child branch to be at thetop of the view, the rule would be that for the last view site on thebranch {right arrow over (R)}={right arrow over (U)}.

The up vectors can change quite drastically along a route to follow theprocedure-specific rules. Using the previously described “always-up”rule, there can be 180 degree of roll between closely-spaced branches.To reflect the fact that endoscopic devices generally move smoothly, theup vectors between branches are linearly blended in a manner similar tothe tangential vectors.

Route-Planning Implementation

Views of the route-planning computer program are shown in FIGS. 6 and 7.In this part of the program, the user can input the various device andprocedure constraints. Finding the acceptable routes within the inputtree as in Methods 1 and 2 is nearly instantaneous. Upon completingthese calculations, the destinations of the closest-routes are displayedand optionally saved to a file. The left half of the applicationdisplays the inputs and outputs for Methods 1 and 2. The user loads acase study which includes the 3D MDCT image, the voxels to which the ROIbelongs, and a paths file containing the view sites of the airway tree.The user inputs the number of closest routes on separate airway-treesegments to be found for each ROI in the “Find Closest” section. The“Geometric Restrictions” section contains the inputs for the device andprocedure constraints. The results of Methods 1 and 2 are displayed inthe “Path to ROI” section. The right half of the application showsdetails related to Method 3. The amount of ROI dilation is entered in“ROI Dilation (mm).” The plot gives a visual representation of thefunctions of each component of the cost function. The user can changethe upper limit on the gray-scale cost (shown at its default of −750)and the relative weighting given to the gray-scale and valley costs. Theresults of the path extension are displayed in FIG. 7. If these resultsare deemed accurate and appropriate for the procedure, the user canappend the extensions to the original paths (centerlines) file.

In the event that the routes determined by Methods 1 and 2 areinadequate, Method 3 is used to extend the existing paths. Method 3usually requires a few seconds (minimum 0.5 sec to maximum 15 sec incases inspected to date) to complete its calculations. After performingpath extensions, the application includes a 3D visualization for viewingthe original paths, ROIs, path extensions, and 3D image data. This isshown in FIG. 7. The user can dynamically interact with the scene bychanging the 3D viewing position and can choose which structures todisplay: the airway tree, any arbitrary slice of the 3D volumetric image(the black and white square), path extensions, or the ROI. By examiningthe different the slices of the 3D medical data through which theextended paths travel, the correctness of extensions is easily verified.The user can use the mouse to look at the scene from any viewinglocation in 3D space. Arbitrary slices of the 3D volumetric image can beoverlayed on the path extensions and ROI so that the user can visuallyinspect the correctness of the route in the volumetric medical image. Ifthe route is incorrect, the user can change parameters in the part ofthe program shown in FIG. 6, including changing the amount of ROIdilation and the branches from which extensions are allowed. Once anacceptable path extension is found, the user can append the extension tothe file containing the centerline locations.

Results

We evaluated the performance of the route-planning methods in 10 humanCT cases. The MDCT data were collected from a Philips Mx8000four-detector scanner and a Siemens Sensation-16 sixteen-detectorscanner. Suspect ROIs, consisting of nodules and infiltrates located inthe lung periphery, were defined by a physician using the 3D live-wiretechnique. A summary of the test cases is given in Table I, with theresults of our automated route-planning method outlined in Table II.

For these results, routes are sought that terminate within 30 mm of thetarget ROI. Method 1 is first used to determine if such routes existwithin the original organ tree T. An example of such a case is given inFIGS. 8A through 8D. In the upper left view shows the route in a surfacerendered view. The route terminates 16.9 mm from the ROI surface. Thesphere shows the route's terminal location with the line segment showingthe viewing direction at this location. The upper right view shows theCT data at this location where the airway diameter is 4.1 mm. Thebottom-left endoluminal view provides a virtual view of the airwaysurface and ROI near the route destination (arrow). The routedestination appears as a fixed-sized arrow in the endoluminal view, theappearance of which, in the 3D perspective view, gives the notion of howfar away the current location is from the destination. The arrow pointsto the ROI center-of-mass, providing an additional 3D cue as to therelative locations of the ROI, current viewing position, and routedestination. The bottom-right view is a weighted-sum coronal projectionof the CT data with the paths, route and the ROI. If no such routeexists, Method 3 or Method 4 finds a path extension. By using thesemethods, acceptable routes are found to every ROI. Method 3 was used toextend the routes in case 21405.3a. Method 4 was used to extend pathsfrom locations with a minimum diameter of 1.5 mm in case b004. FIGS. 9Ato 9C depict a route containing a path extension with a 3D view of theexisting centerline locations, the ROI, the path extension and a sliceof the 3D data through which the extension is defined. The image shownin FIG. 9A is a 3D representation of the centerlines, extension and ROI.The image shown in FIG. 9B is a close-up of the same view. The imageshown in FIG. 9C shows a close-up view of the ROI, centerlines, andextension as well as a slice through which the extension traverses. Thisslice clearly shows an airway that was missed during the initialsegmentation and may provide an adequate route to the ROI. The 3D MDCTimage size is 512×512×596 with Δx=Δy=0.74 mm, Δz=0.60 mm (case h019).FIG. 10 shows an integrated set of views of a path extended to reach aperipheral lung nodule ROI. The 3D MDCT image size is 512×512×414 withΔx=Δy=0.57 mm, Δz=0.60 mm (case h001). The leftmost view shows theextended path in a surface rendered view. Previously existing pathsdisplay both centerlines and a rendering of the segmented airway whilethe extended path only shows the centerline. The middle views showrepresentations of the CT volume at the location on the extended branchshown in the coronal projection view on the right. The middle-top imagedisplays an oblique cross-section of the volume with a viewing directionoriented tangentially to the extended path. The middle-bottom imageshows an endoluminal rendering of the data set as viewed in the samedirection. The length of the path extension is 28.1 mm.

Table I is a summary of the images used in the evaluation of theroute-planning method. ROI location describes the lung lobe (R)ight,(L)eft, (U)pper, (M)iddle, (L)ower in which the ROI is defined.

TABLE I Number of Axial-Plane ROI Patient ID Scanner Slices Resolution(mm) Location b001 Siemens 193 0.55 RUL b004 Siemens 338 0.81 RUL b005Siemens 308 0.73 LLL h001-2 ROIs Philips 414 0.57 RLL h002 Philips 5150.59 LLL h005 Philips 479 0.59 RML h013 Philips 539 0.76 RLL h019Philips 597 0.74 LLL p2h038 Philips 410 0.67 LUL 21405.3a Siemens 7060.67 RML

Table II shows a summary of results of route planning to ROIs located inthe lung periphery using Methods 1 and 3. For those cases where pathextension (Method 3 or Method 4) is necessary, results are shown(before, after) the path extension. Using a 30 mm maximum allowabledistance from the route's terminal location, the method yields adequateroutes to all ROIs investigated (column 3).

TABLE II Distance from Number of ROI in Route Termination Method 3Airway Terminating Airway Patient ID Airway to ROI (mm) Necessary?Generations Diameter (mm) b001 No 16.5 No 5 2.8 b004 No 42.4, 29.2 Yes5, 8 3.3, 1.5 b005 No 18.0 No 6 2.8 h001-ROI 1 No 28.3 No 8 2.1 h001-ROI2 No 17.7 No 9 2.3 h002 Yes 0.0 No 6 3.2 h005 Yes 0.0 No 5 4.8 h013 No12.3 No 6 3.1 h019 No 28.3 No 5 3.7 p2h038 No 22.9 No 8 1.8 21405.3a No35.9, 28.3 Yes 8, 9 2.2, 1.4

The results presented in Table III show how utilizing device constraintsin Method 2 may yield different routes than those previously determined.In the table, the constraint on the minimum path diameter is chosen tobe 2.8 mm in most instances, reflecting the diameter of currentlyavailable ultra-thin bronchoscopes. A diagnostic field of view of either180 degrees, which essentially eliminates this constraint or 70 degrees,a more exacting standard, is required at terminal route locations. Whentaking into account these device constraints, routes do not exist to alltarget ROIs when using the same 30 mm cutoff as before.

TABLE III Endoscope Diagnostic Distance (mm) Number of Minimum AirwayDiameter FOV from Route Airway Diameter Patient ID Input (mm) Input(deg) Termination Generations Encountered (mm) b001 2.8 180 23.1 4 3.5b005 2.8 180 34.7 5 3.6 b005 2.4 70 32.0 6 2.4 h001-ROI 1 2.8 70 69.6 66.1 h001-ROI 1 2.8 180 38.0 9 2.9 h001-ROI 2 2.8 70 16.9 9 4.1 h002 2.870 4.7 6 2.8 h005 2.8 70 1.4 5 2.8 h0013 2.8 70 24.4 6 2.8 h019 2.8 7028.7 5 3.7 p2h038 2.8 70 46.3 5 3.0 p2h038 2.8 180 36.4 6 3.0

FIGS. 11A-11H illustrates how the view directions are modified by Method5 to reach an ROI in the anterior segment of the right upper lobe of thelung. In this example, view directions are chosen so that each viewingsite faces the end of the branch to which it belongs and the next branchalong the route is up (oriented at the top of the view) by the end ofthe current branch. Starting at FIG. 11A, the viewing location is in themain carina. In FIG. 11B, the scene is rotated so the right mainbronchus appears in the top of the view. In FIG. 11C, theright-upper-lobe takeoff is seen in the distance. In FIG. 11D, theright-upper-lobe takeoff has been oriented toward the top of the view.Continuing down the right upper lobe bronchus in FIG. 11E, thetrifurcation into the anterior, posterior, and apical segments is seenin the distance. As the trifurcation gets nearer, in FIG. 11F, the viewis rotated so the posterior segment is toward the top of the view. InFIG. 11G, the ROI and route destination (arrow) are seen in thedistance. In FIG. 11H, the route destination and ROI are nearly reached.The ROI surface is located at a distance of 12 mm from the final routedestination.

REFERENCES

-   1. P. Rogalla, J. Van Scheltinga, and B. Hamm, Virtual Endoscopy and    Related 3D Techniques, Springer-Verlag, Berlin, 2002.-   2. D. Selle, P. Preim, A. Schenk, and H. Peitgen, “Analysis of    vasculature for liver surgical planning,” IEEE Transactions on    Medical Imaging 21(11), pp. 1344-1357, November 2002.-   3. A. Wahle, M. Olszewski, and M. Sonka, “Interactive virtual    endoscopy in coronary arteries based on multimodality fusion,” IEEE    Transactions on Medical Imaging 23(11), pp.-   1391-1403, November 2004.-   4. N. C. Dalrymple, S. R. Prasad, M. W. Freckleton, and K. N.    Chintapalli, “Introduction to the language of three-dimensional    imaging with multidetector CT,” Radiographics 25(5), pp. 1409-1428,    September-October, 2005.-   5. K. P. Wang and A. Mehta, eds., Flexible Bronchoscopy, Blackwell    Science, Cambridge, Mass., 1995.-   6. W. E. Lorensen, F. A. Jolesz, and R. Kikinis, “The exploration of    cross-sectional data with a virtual endoscope,” Interactive    Technology and the New Health Paradigm, pp. 221-230, January 1995.-   7. G. D. Rubin, C. F. Beaulieu, V. Argiro, H. Ringl, A. M.    Norbash, J. F. Feller, M. D. Dake, S. Napel, R. B. Jeffrey, and S.    Napel, “Perspective volume rendering of CT and MR images:    applications for endoscopic imaging,” Radiology 199(2), pp. 321-330,    May 1996.-   8. A. Jemal, R. Tiwari, T. Murray, A. Ghafoor, A. Samuels, E.    Ward, E. Feuer, and M. Thun, “Cancer statistics, 2004,” CA Cancer J.    Clin. 54, pp. 8-29,2004.-   9. J. P. Helferty, A. J. Sherbondy, A. P. Kiraly, and W. E. Higgins,    “System for live virtual-endoscopic guidance of bronchoscopy,” in    IEEE Conf. Computer Vision and Pattern Recognition, 3, pp. 68-75,    20-26 Jun. 2005.-   10. W. E. Higgins, L. Rai, S. A. Merritt, K. Lu, N. T. Linger,    and K. C. Yu, “3D image fusion and guidance for computer-assisted    bronchoscopy,” in SPIE Optics East: Three-Dimensional TV, Video, and    Display IV, B. Javidi, F. Okano, and J.-Y. Son, eds., 6016, pp.    86-100, November 2005.-   11. A. D. Sihoe and A. P. Yim, “Lung cancer staging,” J. Surgical    Research 117(1), pp. 92-106, March 2004.-   12. F. Asano, Y. Matsuno, T. Matsushita, H. Kondo, Y. Saito, A.    Seko, and Y. Ishihara, “Transbronchial Diagnosis of A Pulmonary    Peripheral Small Lesion Usingan Ultrathin Bronchoscope with Virtual    Bronchoscopic Navigation,” Bronchology 9(2), pp. 108-111, April    2002.-   13. B. Geiger, A. P. Kiraly, D. P. Naidich, and C. L. Novak,    “Virtual bronchoscopy of peripheral nodules using arteries as    surrogate pathways,” in SPIE Medical Imaging 2005: Physiology,    Function, and Structure from Medical Images, A. A. Amini and A.    Manduca, eds., 5746, pp. 352-360, 2005.-   14. H. Minami, Y. Ando, F. Nomura, S. Sakai, and K. Shimokata,    “Interbronchoscopist variability in the diagnosis of lung cancer by    flexible bronchoscopy,” Chest 105(2), pp. 1658-1662, June 1994.-   15. D. Aykac, E. A. Hoffman, G. McLennan, and J. M. Reinhardt,    “Segmentation and analysis of the human airway tree from    three-dimensional X-ray CT images,” IEEE Transactions on Medical    Imaging 22(8), pp. 940-950, August 2003.-   16. A. P. Kiraly, W. E. Higgins, E. A. Hoffman, G. McLennan,    and J. M. Reinhardt, “3D human airway segmentation methods for    virtual bronchoscopy,” Academic Radiology 9(10), pp. 1153-1168,    October 2002.-   17. C. Fetita, F. Preteux, C. Beigelman-Aubry, and P. Grenier,    “Pulmonary airways: 3-D reconstruction from multislice CT and    clinical investigation,” IEEE Transactions on Medical Imaging    23(11), pp. 1353-1364, November 2004.-   18. C. Fetita, F. Prêteux, and P. Grenier, “Three-dimensional    reconstruction of the bronchial tree in volumetric computerized    tomography: Application to computerized tomography bronchography,”    Journal of Electronic Imaging 15, pp. 1-17, April/June 2006.-   19. M. S. Hassouna and A. A. Farag, “Robust centerline extraction    framework using level sets,” IEEE Conf. Computer Vision and Pattern    Recognition 1, pp. 458-465, 2005.-   20. A. P. Kiraly, J. P. Helferty, E. A. Hoffman, G. McLennan,    and W. E. Higgins, “3D path planning for virtual bronchoscopy,” IEEE    Trans. Medical Imaging 23(11), pp. 1365-1379, November 2004.-   21. K. C. Yu, E. L. Ritman, and W. E. Higgins, “3D model-based    vasculature analysis using differential geometry,” in IEEE Int.    Symp. on Biomedical Imaging, pp. 177-180, Arlington, Va., 15-18 Apr.    2004.-   22. R. D. Swift, A. P. Kiraly, A. J. Sherbondy, A. L. Austin, E. A.    Hoffman, G. McLennan, and W. E. Higgins, “Automatic axes-generation    for virtual bronchoscopic assessment of major airway obstructions,”    Computerized Medical Imaging and Graphics 26(2), pp. 103-118,    March-April 2002.-   23. M. Wan, Z. Liang, Q. Ke, L. Hong, I. Bitter, and A. Kaufman,    “Automatic centerline extraction for virtual colonoscopy,” IEEE    Trans. Medical Imaging 21(12), pp. 1450-1460, December 2002.-   24. T. Deschamps and L. D. Cohen, “Fast extraction of minimal paths    in 3D images and applications to virtual endoscopy,” Medical Image    Analysis 5, pp. 281-289, 2001.-   25. W. Higgins, S. Ferguson, K. Thomas, J. Helferty, A. Kiraly, A.    Sherbondy, J. Turlington, E. Hoffman, and G. McLennan, “Progress    toward virtual-bronchoscopic guidance of peripheral nodule biopsy,”    Am. J. Respiratory and Critical Care 167(7), p. A535, May 2003.-   26. G. Ferretti, I. Bricault, and M. Coulomb, “Virtual tools for    imaging of the thorax,” Eur. Respir. J. 18, pp. 381-392,2001.-   27. J. Z. Turlington and W. E. Higgins, “New techniques for    efficient sliding thin-slab volume visualization,” IEEE Transactions    in Medical Imaging 20(8), pp. 823-835, August 2001.-   28. A. P. Kiraly, J. M. Reinhardt, E. A. Hoffman, G. McLennan,    and W. Higgins, “Virtual bronchoscopy for quantitative airway    analysis,” SPIE Medical Imaging 2005: Physiology, Function, and    Structure from Medical Images A. Amini and A. Manduca (eds.), v.    5746, pp. 369-383, 2005.-   29. M. Kukuk, B. Geiger, and H. Muller, “TBNA-protocols: guiding    transbronchial needle aspirations without a computer in the    operating room,” MICCAI 2001 W. Niessen and M Viergever (eds.), vol.    LNCS 2208, pp. 997-1006, 2001.-   30. M. Kukuk, “Modeling the internal and external constraints of a    flexible endoscope for calculating its workspace: application in    transbronchial needle aspiration guidance,” SPIE Medical Imaging    2002: Visualization, Image-Guided Procedures, and Display S. K. Mun    (ed.), v. 4681, pp. 539-550, 2002.-   31. M. Kukuk, A Model-Based Approach to Intraoperative Guidance of    Flexible Endoscopy. PhD thesis, University of Dortmund, March 2002.-   32. A. Austin, “Automatic analysis methods for virtual endoscopic    assessment,” Master's thesis, The Pennsylvania State University,    Department of Electrical Engineering, 2000.-   33. K. Mori, S. Ema, T. Kitasaka, Y. Mekada, I. Ide, H. Murase, Y.    Suenaga, H. Takabatake, M. Mori, and H. Natori, “Automated    nomenclature of bronchial branches extracted from CT images and its    application to biopsy path planning in virtual bronchoscopy,” in    Medical Image Computing and Computer-Assisted Intervention, J.    Duncan and G. Gerig, eds., 3750, pp. 854-861, 2005.-   34. P. Heng, P. Fung, T. Wong, Y. Siu, and H. Sun, “Interactive    navigation and bronchial tube tracking in virtual bronchoscopy,” in    Medicine Meets Virtual Reality, J. D. Westwood, ed., (7), pp.    130-133, IOS Press, 1999.-   35. B. Geiger, G. Weiner, K. Schulze, J. Bilger, P. Krebs, K. Wolf,    and T. Albrecht, “Virtual bronchoscopy guidance system for    transbronchial needle aspiration,” in SPIE Medical Imaging 2005:    Physiology, Function, and Structure from Medical Images, A. Amini    and A. Manduca, eds., 5746, pp. 361-368, 2005.-   36. P. Haigron, M. E. Bellemare, O. Acosta, C. Goksu, C. Kulik, K.    Rioual, and A. Lucas, “Depth-map-based scene analysis for active    navigation in virtual angioscopy,” IEEE Transactions on Medical    Imaging 23(11), pp. 1380-1390, November 2004.-   37. S. L. Aquino and D. J. Vining, “Virtual bronchoscopy,” Clinics    in Chest Med. 20(4), pp. 725-730, December 1999.-   38. T. Lee, P. Lin, C. Lin, Y. Sun, and X. Lin, “Interactive 3-D    virtual colonoscopy system,” IEEE Trans. Inform. Tech. Biomed. 3(2),    pp. 139-150, June 1999.-   39. K. Kreeger, F. Dachille, M. Wax, and A. E. Kaufman, “Covering    all clinically significant areas of the colon surface in virtual    colonoscopy,” SPIE Medical Imaging 2002: Physiology and Function    from Multidimensional Images, 4321, pp. 198-206,2002. C. T. Chen    and A. V. Clough (ed.).-   40. K. Mori, Y. Suenaga, and J. Toriwaki, “Fast software-based    volume rendering using multimedia instructions on PC platforms and    its application to virtual endoscopy,” SPIE Medical Imaging 2003:    Physiology and Function: Methods, Systems and Applications, 5031,    pp. 111-122, 2003. A. V. Clough and A. A. Amini (ed.).-   41. D. Kang and J. Ra, “A new path planning algorithm for maximizing    visibility in computed tomography colonography,” IEEE Transactions    on Medical Imaging 24(8), pp. 957-968, August 2005.-   42. T. Fujii, H. Asakura, H. Emoto, N. Sugou, T. Mito, and I.    Shibata, “Automatic path searching for minimally invasive    neurosurgical planning,” SPIE Medical Imaging 2002: Physiology and    Function from Multidimensional Images, S. K. Mun, ed. 4681, Feb.    23-28, 2002.-   43. E. N. Mortensen and W. A. Barrett, “Interactive segmentation    with intelligent scissors,” Graphical Models and Image Processing    60(5), pp. 349-384, 1998.-   44. K. Lu and W. E. Higgins, “Improved 3D live-wire method with    application to 3D CT chest image analysis,” in SPIE Medical Imaging    2006: Image Processing, J. M. Reinhardt and J. P. W. Pluim, eds.,    6144, pp. 189-203, 2006.-   45. T. H. Connen, Introduction to Algorithms, MIT Press, Cambridge,    Mass., 2001.-   46. R. C. Gonzalez and R. E. Woods, Digital Image Processing,    Addison Wesley, Reading, Mass., 2nd. ed., 2002.-   47. K. Mori, J. Hasegawa, Y. Suenaga, and J. Toriwaki, “Automated    anatomical labeling of the bronchial branch and its application to    the virtual bronchoscopy system,” IEEE Trans. Medical Imaging 19(2),    pp. 103-114, February 2000.-   48. J. Tschirren, G. McLennan, K. Palagyi, E. A. Hoffman, and M.    Sonka, “Matching and anatomical labeling of human airway tree,” IEEE    Trans. Medical Imaging 24(12), pp. 1540-1547, December 2005.

We claim:
 1. A method of planning a route through a tubular organ,comprising the steps of: generating a 3D image of a tubular organ to benavigated during an actual, follow-on endoscopic procedure using anendoscopic device; wherein the generated image of the tubular organincludes a target region of interest (ROI); receiving constraintinformation regarding one or more of the following: the anatomy of thetubular organ, the structure or function of the endoscopic device, andcharacteristics of the endoscopic procedure; automatically generating atleast one optimized route to be used by the endoscopic device to reachthe ROI during the follow-on endoscopic procedure; wherein the optimizedroute utilizes the constraint information; and displaying the optimizedroute in conjunction with the 3D image of a tubular organ.
 2. The methodof claim 1, including the step of extending an optimized route, ifnecessary, to reach an ROI outside of the tubular organ.
 3. The methodof claim 1, including the step of modifying the viewing direction ateach site along a route to give physically meaningful navigationdirections or to reflect the requirements of a follow-on live endoscopicprocedure.
 4. The method of claim 1, wherein the information includesanatomical constraints that define locations or organs to avoid.
 5. Themethod of claim 1, wherein the information includes anatomicalconstraints that confine the route within specific geometric locations.6. The method of claim 1, wherein the information includes a metric forselecting the most appropriate route.
 7. The method of claim 6, whereinthe metric is the closest route to the ROI such that the route satisfiesall applicable anatomical, device, and procedural constraints.
 8. Themethod of claim 1, wherein the information includes the definition ofthe ROI.
 9. The method of claim 1, wherein the information includes asegmentation of the organ through which navigation will occur in eitherthe 3D image or in the real organ with an endoscopic device.
 10. Themethod of claim 1, wherein the information includes the central axes ofthe segmented organ.
 11. The method of claim 1, wherein the informationincludes a parametric description of the endoscopic device.
 12. Themethod of claim 11, wherein the parametric description includes thediameter, flexibility, or other physical characteristics of theendoscopic device.
 13. The method of claim 11, wherein the parametricdescription includes descriptions of ancillary devices that may be usedin conjunction with the primary endoscopic device.
 14. The method ofclaim 1, wherein the organ is a branching organ.
 15. The method of claim1, wherein the organ is an airway tree.
 16. The method of claim 1,wherein the information is derived through a multidetector computedtomographic (MDCT) chest image.
 17. A method of planning an endoscopicroute through a tubular organ to a target diagnostic region of interest(ROI), comprising the steps of: providing information about a tubularorgan and an actual, follow-on endoscopic procedure associated with theorgan; automatically identifying the most appropriate route or routes tothe ROI within the organ in accordance with the information provided;extending the route beyond the confines of the tubular organ if the ROIlies outside of the tubular organ; and displaying the route or routesincluding any route extensions in conjunction with the 3D image of thetubular organ.
 18. The method of claim 17, wherein the informationincludes anatomical constraints that define locations or organs toavoid.
 19. The method of claim 17, wherein the information includesanatomical constraints that confine the route within specific geometriclocations.
 20. The method of claim 17, wherein the information includesa metric for selecting the most appropriate route.
 21. The method ofclaim 20, where the metric is the closest route to the ROT such that theroute satisfies all applicable anatomical, device, and proceduralconstraints.
 22. The method of claim 17, wherein the informationincludes the definition of the ROI.
 23. The method of claim 17, whereinthe information includes a segmentation of the organ through which theendoscopic device will navigate.
 24. The method of claim 17, wherein theinformation includes central axes of the segmented organ.
 25. The methodof claim 17, wherein the information includes a parametric descriptionof the endoscopic device.
 26. The method of claim 25, wherein theparametric description includes the diameter, flexibility, or otherphysical characteristics of the endoscopic device.
 27. The method ofclaim 25, wherein the parametric description includes descriptions ofancillary devices that may be used in conjunction with the primaryendoscopic device.
 28. The method of claim 17, wherein the organ is abranching organ.
 29. The method of claim 17, wherein the organ is anairway tree.
 30. The method of claim 17, wherein the information isderived through a multidetector computed tomographic (MDCT) chest image.31. A system for planning routes through a tubular organ, comprising: adisplay device for displaying a 3D image of a tubular organ to benavigated during an actual, follow-on endoscopic procedure using anendoscopic device; a memory for storing information about the organ andthe follow-on endoscopic procedure, the information including constraintinformation regarding one or more of the following: the anatomy of thetubular organ, the structure or function of the endoscopic device, andcharacteristics of the endoscopic procedure; and a processor operativeto perform the following functions: a) access the stored information andautomatically generate at least one optimized route to be used by theendoscopic device to reach the ROI during the follow-on endoscopicprocedure, and b) display the route or routes on the display device inconjunction with the image of the tubular organ.
 32. The system of claim31, wherein the processor is further operative to determine a routebeyond the organ.
 33. The system of claim 31, wherein the processor isfurther operative to modify the viewing direction at each site along aroute to give physically meaningful navigation directions or to reflectthe requirements of a follow-on live endoscopic procedure.
 34. Thesystem of claim 31 wherein the stored information includes anatomicalconstraints that define locations or organs to avoid.
 35. The system ofclaim 31, wherein the stored information includes anatomical constraintsthat confine the route within specific geometric locations.
 36. Thesystem of claim 31, wherein the stored information includes a metric forselecting the most appropriate route.
 37. The system of claim 36,wherein the metric is the closest route to the ROI such that the routesatisfies all applicable anatomical, device, and procedural constraints.38. The system of claim 31, wherein the stored information includes thedefinition of the ROI.
 39. The system of claim 31, wherein the storedinformation includes a segmentation of the organ through whichnavigation will occur in either the 3D image or in the real organ withan endoscopic device.
 40. The system of claim 31, wherein the storedinformation includes the central axes of the segmented organ.
 41. Thesystem of claim 31, wherein the stored information includes a parametricdescription of the endoscopic device.
 42. The system of claim 41,wherein the parametric description includes the diameter, flexibility,or other physical characteristics of the endoscopic device.
 43. Thesystem of claim 41, wherein the parametric description includesdescriptions of ancillary devices that may be used in conjunction withthe primary endoscopic device.
 44. The system of claim 31, wherein thestored information is derived through a multidetector computedtomographic (MDCT) chest image.